feat(voice): GPU-less client-side speaker-embedding module (audit H3, flag-gated default OFF)#224
Merged
Conversation
… flag-gated default OFF)
VOICE twin of the face client-embedding path. When VITE_CLIENT_SIDE_VOICE_EMBEDDING
is ON, the browser computes the 256-d Resemblyzer speaker embedding locally
(onnxruntime-web, WASM EP) and uploads ONLY the vector — the raw audio never
leaves the device. Default OFF = byte-identical legacy audio upload.
- src/features/biometrics/voice-embedding/: clientVoiceEmbeddingFlag.ts (mirrors
clientEmbeddingFlag), voicePreprocess.ts (WAV decode → dBFS normalize → mel
partials + partial slicing), speakerEmbedder.ts (SpeakerEmbedder ONNX class,
lazy dynamic import, reuses embedding/modelCache.getModel; mean-of-partials +
L2-norm = embed_utterance), embedCapturedVoice.ts (orchestrator, number[256] |
null, never throws), prefetchVoiceModel.ts (mirrors prefetchFacenetModel).
- MfaStepRenderer VOICE case: flag ON → verifyStep(VOICE, { embedding }); on ANY
failure FALL BACK to { voiceData } (the preprocessing is scaffold + the server
accepts either when ON, so a gap never blocks login — unlike FACE which
hard-fails). VoiceStep mount schedules the model prefetch (flag-gated no-op).
- Model fetched at RUNTIME only when ON; NOT added to public/models/manifest.json
(a build-prefetch entry would FATAL fetch-models, the facenet failure). FP32.
- i18n: mfa.voice.preparingSecure added to en.json + tr.json.
SCAFFOLD — NOT parity-validated: the ONNX export + inference are exact
(torch<->ONNX cosine 1.0), but the JS preprocessing does NOT reproduce
Resemblyzer's WebRTC-VAD silence trim (no bit-exact JS port) and uses a
librosa-approximate mel — skipping the VAD shifts ≈0.11 cosine vs server (risky
near the 0.65 threshold). KEEP THE FLAG OFF until the client mel+VAD are validated.
Contract + canary steps: biometric-processor docs/design/VOICE_CLIENT_EMBEDDING_SPEC.md.
Tests (vitest): voicePreprocess (WAV decode round-trip, dBFS normalize, mel shape,
partial slicing), speakerEmbedder (per-partial (1,160,40) feed, mean+L2-norm,
session reuse — fake ort), clientVoiceEmbeddingFlag, MfaStepRendererVoiceEmbedding
(OFF→voiceData, ON→embedding[256], ON+fail→fallback to voiceData). 37 tests across
9 files green (incl. the existing face renderer tests unbroken); tsc + eslint clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Audit item H3 (web-app side). VOICE twin of the face client-embedding path: when
VITE_CLIENT_SIDE_VOICE_EMBEDDINGis ON, the browser computes the 256-d Resemblyzer speaker embedding locally (onnxruntime-web, WASM EP) and uploads only the vector — the raw audio never leaves the device. Default OFF = byte-identical legacy audio upload ({ voiceData }).What's added
src/features/biometrics/voice-embedding/—clientVoiceEmbeddingFlag.ts(mirrorsclientEmbeddingFlag.ts),voicePreprocess.ts(WAV decode → dBFS normalize → mel partials + partial slicing),speakerEmbedder.ts(SpeakerEmbedderONNX class — lazyimport('onnxruntime-web'), reusesembedding/modelCache.getModel; mean-of-partials + L2-norm == Resemblyzerembed_utterance),embedCapturedVoice.ts(orchestrator,number[256] | null, never throws),prefetchVoiceModel.ts(mirrorsprefetchFacenetModel).MfaStepRenderer.tsxVOICE case — flag ON →verifyStep(VOICE, { embedding }); on ANY failure FALL BACK to{ voiceData }(unlike FACE which hard-fails — the voice preprocessing is scaffold + the server accepts either when its flag is ON, so a model/preproc gap never blocks login).VoiceStepmount schedules the model prefetch (flag-gated no-op).public/models/manifest.json(a build-prefetch entry would FATALfetch-models/deploy — the exact facenet failure). FP32 ships (INT8 rejected on ort-web WASM).mfa.voice.preparingSecureadded toen.json+tr.json.The ONNX export + inference are exact (torch↔ONNX cosine 1.0, verified in the biometric-processor PR), but the JS preprocessing is NOT parity-validated: it does not reproduce Resemblyzer's WebRTC-VAD silence trim (no bit-exact JS port) and uses a librosa-approximate mel. Skipping the VAD measured ≈0.11 cosine shift vs the server on clean audio — risky near the 0.65 accept threshold. Keep
VITE_CLIENT_SIDE_VOICE_EMBEDDINGOFF until the client mel+VAD are validated to parity. Full contract + the exact validation/canary steps: biometric-processordocs/design/VOICE_CLIENT_EMBEDDING_SPEC.md.Tests
vitest:
voicePreprocess(WAV round-trip, dBFS normalize, mel shape, partial slicing),speakerEmbedder(per-partial(1,160,40)feed, mean+L2-norm, session reuse — fakeort),clientVoiceEmbeddingFlag,MfaStepRendererVoiceEmbedding(OFF→voiceData, ON→embedding[256], ON+fail→fallback tovoiceData). 37 tests across 9 files green (incl. the existing face renderer tests unbroken).tsc --noEmit+ eslint clean.Paired PRs: biometric-processor (endpoints + ONNX export, VERIFIED) + identity-core-api (voice policy + handler routing + compose passthrough).